On efficiency of Orthogonal Matching Pursuit .0 

Eugene Livshit 
Abstract 

We show that if a matrix <I> satisfies the RIP of order [C-fT^'^] with isometry 
constant 6 = cK~^'^ and has coherence less than 1/{20K^'^), then Orthogonal 
Matching Pursuit (OMP) will recover i^-sparse signal x from y = in at most 
[CX^'^J iterations. This result implies that X-sparse signal can be recovered via 
OMP by M = ©(K^-^logiV) measurements. 

1 Introduction. 

The emerging theory of Compressed Sensing (CS) has provided a new framework for signal 
acquisition [1], [3], [8]. Let us recall some basic concepts of CS. Let $ be a M x matrix 
(M < N). The basic problem in CS is to construct a stable and fast algorithm for recovery 
a signal x e M'' that has K non-zero components (i^-sparse signal) from measurements 
y = $x G M*^ and to determine (M, N, K) for which such algorithms exist. 
E. Candes and T. Tao proved that Basic Pursuit (BP) 

x{y) = argmin{|2;|i : = y}. 

can provide the exact recovery of arbitrary K-sparse x G by M = 0(i^'log(A^/-ft')) 
measurements. 

In this article we study signal recovery via Orthogonal Matching Pursuit (OMP). Al- 
though theoretical results for OMP are essentially worse than for BP, its computational 
simplicity allows OMP to achieve very good result in practise [18]. 



Algorithm: Orthogonal Matching Pursuit 



Input: $, y. 

Initialization: r° := y, x^ := 0, A° = 0, / = 0. 
Iterations: Define 

j^i+i .= U argmaXj|(r' 



x'+i := argmin^^ supp(^)cA'+i Wv " "^^ll ^'^^ ■= V - 

If r^+i = 0, stop. Otherwise let / := / + 1 and begin a new iteration. 
Output: If algorithm stops at l-th iteration, output is x = Xi. 



By (pi, 1 < i < N, we denote the i-th column of $. We assume that = 1, 

1 < i < N. To formulate results on recovery via OMP we use two basic properties of 
matrix $. 
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• Coherence of $ 

:= sup 

• Restricted Isometry Property (iij). A matrix $ satisfies Restricted Isometry Prop- 
erty (RIP) of order K with isometry constant 5 G (0, 1) if the inequahty 

< \\<^xf < {l + 6)\\xf 

holds for all /^-sparse x G M^. 
It's well known (see [II], [12]) that if 

then OMP will recover arbitrary i^-sparse signal x from y = $x in exactly K iterartions. 
The stability of recovery via OMP in the term of coherence of $ has been studied in 
[IT], [H], [9], [10], [n], [ig. Recently M. Davenport and M. Wakin [6], and E. Liu and 
V.N. Temlyakov [H] showed that if $ satisfies RIP of order K + 1 with isometry constant 

then OMP recovers arbitrary if-sparse signal x G in exactly K iterations. 

To compare these results we recall estimates on coherence and RIP for normalized 
random Bernoulli matrices $ (each entry is ±M~^/^ with probability 1/2). For rather big 
we have with high probability that 

R. Baraniuk, M. Davenport R. Devore and M. Wakin [2] (see also earlier B.S. Kashin's 
work showed that random Bernoulli matrix $ with high probability satisfy RIP of 
order K with isometry constant 6 for 

M^O(^^M^). (2) 

Thus both results require M = 0{K'^) measurements for recovery of i^-sparse signal. The 
aim of this article to show that OMP can recover sparse signals by essentially less number 
of measurements. 

Theorem 1. There exist absolute constants C > and c > such that if $ satisfies 
the RIP of order [CK^''^] with isometry constant 5 = cK~^ '^ and has coherence /i($) < 
1/{20K^'^), then for any K-sparse x G M^, OMP will recover x exactly from y = $x in 
at most [CK^''^] iterations. 

Inequalities ([1]) and ([2]) imply that for rather big absolute constant Cm > with high 
probability normalized random Bernoulli matrix $ with 

M= [Cm K^-"^ log N] 
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satisfies condition of Theorem [TJ 

Mucli less is known about the lower estimates. H. Rauhut proves that if M < cK^^'^ 
then for most random M x N matrices there exists a ii'-sparse signal x G that can 
not be recovered via K iterations of OMP. Moreover, it's conjectured in [16] (see also 
[5]) that for M < CnK'^~^^", n E N, with high probability there exists a i^-sparse signal 
X G that can not be recovered via K iterations of OMP from y = ^x. 



2 Auxiliary lemmas. 

We use two results on the rate of convergence of Orthogonal Greedy Algorithm (OMP). 

Theorem A. (R.A. Devore, V.N. Temlyakov, [7J) Suppose that y = Then 
for any I > 1 we have 



< \xUl-'/'. 



Theorem B. (EL, [15]) For anyl,l<l< l/(20/i($)) we have 

llr^'ll <3ai(y,$). 



For / > we set 



:= X - xK 



Then by definition of OMP 

= y - $x' = <l>x - $a;' = / > 0. (3) 

Assume that 

X = {xi, xn), = {z[, . . . , 2;jv)> l>0. 

Set 

Vo = suppx, tlK) < K. (4) 

By x\v, V C Vo, denote an element (xi, . . . ,x„) of such that Xi = Xi, i E V, and 
Xi = 0, i ^ V. For each C Vq we define 



Let 

C := 2 X 10^ c := 10~^ 5 := cR-^-^. (5) 
Lemma 1. Suppose that I + K < CK^'^. Then we have 

Y,i4r<3mvo\A'), (6) 

R{Vo\A')<{l + 26)\\r'f. (7) 
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Proof. It's clear that \z^\o < \x\o + |x'|o < K + 1 < CK^ '^, so by RIP and ([3D we have 

N N 

(1 - s) < ii$^'f = \\rr <{i+s) J2{zir. (s) 
1=1 1=1 

On the other hand, using definition of R{-), and RIP for x\y^^\/^i we write 

\\x\v,\A4' = RiVo\A'), 

(1 - 6)R{Vo \ A') < 11$ {x\y^\j,.) II' < (1 + 6)R{Vo \ A')- (9) 
The definition of OMP implies that 

\\^zr = \\rr<\\<^{xWAA')\\"- 
Therefore using ([8]) and ([9]) we have 

N 

(1 - J2{zir < IklP < 11$ (x|v,\aO ir < (1 + WVo \ A'), 

i=l 

\ieA' ieVo\A' J 

J2i^D' + R{Vo\A')<l^^R{Vo\A'). 

J2{4r < ([^ - miVo \ AO < 35i?(Fo \ AO. 
This completes the proof of From it follows that 

N 

R{Vo \ AO = 5^ (^D' < ^ (1 - ^ (1 + 2^)ll^'ll'- 

□ 

For increasing sequence = /q < /i < • • ■ < /s, s > 1, we denote 

Vk := Vo \ A'^ Rk = RiVk), 0<k<s. (10) 
Lemma 2. Suppose that h + K < CK^ "^ , 1 < k < s. Then for arbitrary p G N we have 



4 



Proof. Since r'*" = ^z^'' we estimate by Theorem A. 

llr'^+Pf < (11) 
P 

So to prove the lemma it's sufficient to estimate 

V *=i / \iGVbuA'fc / \ \ieVo\A'k J \jeA'fe 

Applying f lTOj) and ([1]) we have 

^iGVb\A'fc / \ieVo\A'^k J \ieVk / i&Vk 

(12) 

Using (|6]) from Lemma [1] we get 



ieA'fc / ieA'-k iGA'fc 

Combining with ( IT2l) we obtain the desirable inequality 

k'N? < RkiGSCK^-^ + 2K). 
This together with ( fTTl) completes the proof of the lemma. □ 

Lemma 3. Lei 1 < p < K^-^ and h + 2p < CK^-^, 1 < k < s. Then for any W C Vk 
such that = p we have 

R{Vk \ A^'=+2f) < 10i?(Vfc \W) + 30SRk. 

Proof. According to RIP, ([3]), (jH]) and flTU]) we estimate 

(ap(r''=))' < ||r''= - Hx\w)f = ||$(^''=) - $(^'NH^)ir = 

l<i<Af, ii^W 

< (1 + 5) ( E (-l^)' + E < (1 + ^) ( E (^^)' + E ('I'n < 

\ieVk\W l<i<N, f^Vk ) \i(iVk\W ieA'k J 

< (1 + 6){R{Vk \ W) + 36R{Vo \ A''^)) < (1 + 6){R{Vk \ W) + 36Rk). 

Since 

p < K^-^ < l/(20/i($)) 



5 



we can apply Theorem B and get 

Using ([7j) from Lemma [1] we obtain 

\ A''^-+'*') = \ A''+'^) < (l + 25)||r'^+2pf < (l + 25)9((Tp(r'^))' < 

< (1 + 25)9(1 + 5)(i?(Vfc \W) + 36Rk) < 10 {R{Vk \W) + 36Rk) 



□ 



3 Proof of Theorem [T]. 

We prove by induction on k that if Ik + K < CK^'^ and Vk 7^ 0, then we can define 
Ik+i > h, Vfc+i and Rk+i satisfying ( ITOj) such that 

4+1 + K< CK^ \ (13) 

and at least one of the following statements hold 

(A) - /fc < 3 X l()\Q5CK^-^ + 2K), i{Vk \ Vk+i) > K^-'; (14) 

(B) 4+i-4<2K°-«, m\Vk+i)>K'-''; (15) 

(C) 4+1 - 4 < 2^0-6, tj(F, \ \4+i) > K'-^; (16) 

(D) /fc+i - 4 < 2K'-\ m \ Vk+i) > K^-'; (17) 
(E) 4+1-4 <2ir°■^ tl(V^AVfc+i)>l. (18) 

Set 

Pa = [3 X 10^(65OT^-^ + 2K)], WA = Vkn A'fe+P^. 
If flW^ > K^-^ we define 

4+1 := 4 + PA, V^+i := Vo \ A''^+^ = Vk \ Wa, Rk+i := R{Vk+i). 

Then ^(Vfc \ V^+i) = tl^A and statement (fT^ holds. The inequality 

Ik+PA + K <CK^-^ 

will be checked below. Assume that 

^Wa < K^-^ (19) 

Applying Lemma |2] we have 

, ^ Rk{65CK^-' + 2K) ^ Rk 

" " - [3 X 10^{66CK^-^ + 2K)] " 2.5 x lO^' 
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Using ([7j) from Lemma [T] we estimate 



Rk ^ Rk 



R{Vk \ Wa) = R{Vo \ A'''+^-) < (1 + 25)||r''=+^-f < (1 + 25)^^^^ < (20) 
Set 

If ^Wb > K^-^ we set 

h+i = lk + 2pB, Vk+i = Vk\A''+' = Vk\WB, Rk+i = R{Vk+i). 

Then tl(Vfc \ Vk+i) = ^Wb > and taking into account ^ we obtain The 

inequahty 

lk + 2pB + K < CK^ '^ 

will be checked below. 
Assume that 

^Wb < K^-^. (21) 
Applying Lemma [3] for W = Wa and p = Pb, and inequality (|20|) we get 

i?(V^fc \ Wb) = R{Vk \ A'^+^P^) < 10R{Vk \ Wa) + 306Rk < ^^j^ + 306Rk. (22) 

We repeat these calculations three more times. 
Set 

Pc:=^Wb, Wc = VknA'''+^P^. 

If ^Wc > K^-^ we set 

4+1 = h + 2pc, Vk+i = \ A'^^^ = \ Wc, Rk+i = RiYu+i). 

Then tl(Vfc \ I4+1) = pVc > K^'^ and taking into account (EI]) we obtain ([16]). The 
inequality 

Ik + ^pc + KK CK^-^ 

will be checked below. 
Assume that 

mc < K'-\ (23) 
Applying Lemma |3] for W = Wb and p = pc, and inequality fl22]) we get 

R{Vk\Wc) = RiVkXA'^+^P'^) < WR{Vk\WB)+3QSRk < —^ + 3(10^ + 10)5/?^. (24) 

X J.U 

Set 

PD:=mc, W^fl = VfcnA''=+2pi5. 

If ftW^B > K^-^ we set 

4+i = 4 + 2p,3, \4_,i = l4\A''=+i = \4\iyB, Rk+i = R{Vk+i). 



Then \^{Vk \ Vk+i) = ^Wd > K^''^ and taking into account ([23]) we obtain ([17]). The 
inequahty 

h + 2pD + K< CK^-^ 

will be checked below. 
Assume that 

< (25) 
Applying Lemma |3] for W = Wc and p = Pd, and inequahty we get 

R{Vk\WD) = i?(\4\A'^+'^^) < lORiVk\Wc)+30SRk < ^ + 3(10^ + 10^ + 10) 5 Rk- (26) 
Set 

PE:=mD, WE = VknA'''+^^^ 
lk+i = lk + 2pE, Vk+i = Vk\A'''+' = Vk\WE, Rk+i = R{Vk+i). 
Taking into account ( l25l) we get 

h+i — h < 2K^''^. 

The inequality 

lk + 2pE + K< CK'-^ 

will be checked below. 

Applying Lemma [3] for W = Wd and p = Pe, and inequalities (l26l) and ([5]) we have 

= R{Vk \ We) = R{Vk \ A'^+^P^) < WR{Vk \ Wd) + 306Rk < 
R 

<Y + ^^^^^ + + + < ^fc(l/2 + 4 X lOH) < Rk. 

Therefore We and tl(Vfc \ Vfc+i) > 1 and statement ([H]) holds. 

Thus to complete the proof of induction assumption it remains to estimate 

h+PA + K, lk + 2pB + K, lk + 2pB + K, lk + 2pc + K, lk + 2pD + K. 

It's clear that the biggest of these numbers is the first one. 

Using induction assumption, inclusion Vq D Vi D ■ ■ ■ D V^, and the equality jjVo = K 
we claim that statement f lT^ (case A) can be fulfilled not more than = K/ K^-^ times, 
statement f[T^ (case B) not more than K^-^ = K/K^-^ times, statement f lTB]) (case C) not 
more than K^-^ = K/K^-^ times, statement f[T7)) (case D) not more than K^-^ = K/K^-"^ 
times, and statement ( IT6]) (case E) not more than K times. Hence, using ([5]), we obtain 

4 < 3 X 1{)\Q5CK^-^ + 2K)K0-2 + 2K°-«K0-^ + 2K''-^K''-^ + 2K°-^K°-^ + 2K°-^K = 

= (18 X lO^SCK^-^ + 6 X 10^ + 8)K^-\ 



lk+PA + K< {18 X 10^5CK^-^ + 6 X 10^ + 8)K^-^ + 3 x 10\Q5CK^-^ + 2K) + K < 

< (36 X lOHCK^-^ + 12 X lO'* + 9)K^-^ < 
< (36 X 10^ X 10"^i^"°-^2 X 10^K°-2 + 12 X 10^ + 9)K^ '^ < 20 x lO^K^ '^ = CK^'^. 
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Since i{Vk \ Vk+i) > 1 there exists s G N such that = and + < CK^-^. 
Therefore by (fTUj) and (E]) we have 

^0 C A'^ 

and 

Using RIP we finally obtain that 

X = □ 

Remark 1 . We guess that constant 3 in Theorem B is not optimal and hence constants C 
and from ([5]) can be reduced. 

The author thanks professor V.N. Temlyakov and professor S.V. Konyagin for useful 
discussions. 
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